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ABSTRACT 

A literature review examined major trends in 
organizations and training and the ways in which evaluation may help 
drive training impact. In addition, 3 case studies and 18 interviews 
were conducted in the United States to study evaluation methods. 
Recommendations about the use of evaluation for training impact were 
then formulated, and four Dutch experts on evaluation and training 
were asked to react on the recommendations in the Dutch context. 
Characteristics identified in the literature were illustrated with an 
example from one case study of a company that sells medical products 
by telephone. The following steps were essential: (1) conduct a needs 
assessment; (2) design and deliver training iteratively; (3) get the 
supervisor and employee together before the training to discuss it: 
(4) follow up on the training; and (5) store evaluation information 
in an electronic database. In contrast to the opinions of U.S. 
evaluators, Dutch experts believed that the training consultant 
should not be responsible for the involvement of supervisors, and 
further indicated that evaluation should be used sparingly by the 
training consultant because it is often seen as threatening in the 
Dutch context. Reasons for these differing perceptions are discussed. 
Four tables present study findings. (Contains 13 references.) 
(SLD) 
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1. Introduction 



This s idy started from the belief that evaluation should not only investigate the impact of 
trainin \ programs, but should also help improve training impact by being an integral part 
of the training development process. Evaluation activities that are undertaken early in the 
design process may yield information about problem areas and may help solve these 
problems early cn. Evaluation during and after the training event may direct and also 
increase the transfer of training and program improvement. 

Starting from this assumption, a literature study was carried out, resulting in a 
review report. The focus in this literature review was on major trends in organizations 
and training, and on ways in which evaluation may help drive training impact in view of 
these trends. 

Many of the suggestions that were found in literature were on the level of ideas 
and assumptions rather than practical experiences an1 findings validated by research. 
Therefore, a qualitative study of exemplary evaluation practices was carried out in the 
USA, to get more concrete how-to-do examples. By means of three in-depth case studies 
of ongoing projects and interviews with 18 experts in the field of Human Resource 
Development, examples of sophisticated uses of evaluation were gathered. Special 
attention was paid to evaluation methods, the problems that were encountered and the 
ways in which they were solved. 

The data have been analyzed with Hyperqual, a computer program for qualitative 
data analysis. In the analysis process, the following steps were undertaken: reviewing 
interview reports; formulating categories; formulating code words for each category; 
coding each interview report on paper; entering code words into the computer; selecting 
and printing information per code word; selecting and combining the most essential 
categories; writing the report. 

Based on the analysis, recommendations about the use of evaluation for training 
impact were formulated. Four Dutch experts on evaluation and training were asked to 
react on the recommendations with comments from their point of view and with 
elaborations for the Dutch context. 

Table 1 presents the research activities, methods and outcomes in chronological 

order. 



3 



Project parts 
Jchronologicall 


Activities 


Approach 


Outcomes 


Literature 
review 


Literature search and 
analysis 


Search in ERIC; PICA, 
Journals, SSCI, etc. 


Literature review: 
Processes that are 
supposed to increase 
training impact 

Resulted in: 

Research questions for the 
empirical part 


American 
part 


- 18 interviews 

- 3 case studies 


Partly structured, mainly 
unstructured 

In-depth, triangulation of 
methods! document 
analysis, interviews, 
observations. 


Practical information m 
successful evaluation 
methods. Contextual 
information on 
respondents, problems, 
solutions, etc. 




Information analysis 


Computer program for 
qualitative data analysis: 
Hyperqual 


Categories of factors that 
increase training impact, 
the role of evaluation in 
this process 

Resulted in: 

Initial recommendations 


Dutch 
part 


4 interviews 


Guided by initial 
recommendations 


Comments on 
recommendations, 
suggestions for 
improvement 

Kesuiieu in. 

Final recommendations 



Table 1 : Research activities, methods and outcomes. 



The structure of this paper is as follows: Chapter 2 starts with the questions that guided 
the literature search and analysis. Following, the main conclusions from the literature 
review are summarized. Chapter 3 starts with the questions that guided the case study 
research and interviews on exemplary evaluation practices in the US. Next, a short 
description is given of one of the case studies. The main part of the chapter deals with 
some of the main findings. Each finding will be illustrated with information from this 
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case study. In chapter 4, the findings will be discussed in relation to the Dutch 
comments. 

2. Literature review 

2.1 Questions 

One of the assertions of the study was that training and evaluation exist in a social context 
and, thus, should not be investigated in isolation. In the literature review, therefore, 
attention was also paid to the organizational or business context in which training and 
evaluation exist. 

The questions that were formulated for this literature study were: 

- What trends occur in organizations and their environment? 

- What kind of trends occur in the field of Human Resource Development to cope with 
these organizational trends? 

- In the light of these trends, how can evaluation help realize and improve the impact of 
training programs? 

2.2 Findings 

For the purpose of the study, training was defined as: "a provision that is aimed at 
creating learning processes that contribute directly or indirectly to the performance of 
employees and the organization" (Voerman, 1993). Training is increasingly seen and 
used as a strategy to deal with organizational trends and problems and to add to 
organizational success. In alignment with this view on training, increasing attention is 
given to what impact it is delivering. 

In the study, evaluation was defined as "a systematic study into training contexts, 
needs, plans, operations, and effects, in order to assist the training professional and his 
clients in improving the value of the program and the utility of later evaluation" 
(Voerman, 1993). Although evaluation is often used to assess the impact of training 
programs, it was the assumption of the study that evaluation can also be very useful in 
driving the impact. 

In the light of the current trends and challenges, literature posited four 
characteristics of training for impact, to which evaluation could add positively. In the 
following paragraphs, each of the four characteristics will be discussed briefly and 
followed by some evaluation questions that can be asked in different stages of the training 
development process: before, during, and after the training event(s) . 



2.2.1 Integration of organizational needs, mission, and goals 

Training quality is not longer judged on the basis of the quality of the materials or direct 
learning results, but on its contribution to organizational goals. Or, as Kaufman and 
Parry (1991) put it: "Good training is good for business - but only if it addresses 
organizational needs and is linked to the organization's business plan". A training that 
yields some effects but that was not needed, is not efficient for the organization (e.g. 
Boydell, 1976; Brinkerhoff, 1989; Garavan, 1991; Jacobson & Kaye, 1991; Kaufman & 
Parry, 1991; Romiszowski, 1981). 

Identifying organizational needs and deriving the training goals from it, is an 
activity that needs to take place in the beginning stage of the design process. However, 
also during and after training delivery, there should be a constant awareness of the 
organizational needs, because needs may change in the course of time (e.g. Brinkerhoff, 
1987; Minick & Medlin, 1983). 

Examples of evaluation questions are given in table 2: 



Before 


During 


After 


- What are the organizational 


- Have goals changed during the 


- To what degree has the 


goals? 


design process? 


original goal been reached? 


- What business units are 


- Is the training addressing 


~ Are further solutions needed? 


concerned with reaching the 


organizational goals? 


- To what degree are parts of the 


goals? 


- Are organizational goals 


organization being affected by 


- What factors are hindering the 


likely going to be reached? 


the training as planned? 


realization of the goals? 


- Are other interventions than 


- Are there any unanticipated 


- How desirable is it to reach 


training carried out at the right 


effects on different parts of the 


the goals? 


moment? 


organization? 


- What are the costs of not 




- What can we learn from it for 


reaching goals? 




the future? 


- What are the likely costs of 






realizing the goals? 






- What are the costs of different 






alternative interventions? 






- What units will be affected if 






the goals will be reached? 






- What ,iterventions are likely 






to solve ihe problem? 






- How do interventions relate to 






each other in realizing the 






goals? 







Table 2: Evaluation questions at certain moments in the training development process to relate training 
to organizational mission, needs, and goals. 
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2.2.2 Anticipation of trainee characteristics and needs 

Based on an extensive research on transfer promoting factors, Baldwin and Ford (1988) 
concluded that several trainee characteristics influence the transfer of training, such as: 
motivation, high job-involvement, and confidence in successful completion of the 
training program. Parry (1990) also mentioned the trainees 1 ability to learn and his 
attention to be important. The individual characteristics of trainees should be identified in 
the training development process, especially because the 'traditional training population' 
is changing rapidly, due to the growing diversity in races and a changed ratio of men and 
women working in the organization. 

Generally, however, it was reported that, nowadays, the focus of the assessment 
should go beyond the personal characteristics of trainees. As the training should have 
impact on an organizational level, it is regarded as important to yield information about 
work related problems and needs as about personal ones. 

Evaluation questions in this respect are given in table 3: 



Before 


During 


After 


- What aspects of the task are 


- Are trainees the same (kind of) 


- Are the trainees applying the 


trainees having trouble with? 


people as the training has 


new skills? 


- Is it a performance problem? 


been planned for? 


- Have problems been solved? 


- To what degree is the 


- Is the training attractive to 


- If not, how can redirection be 


performance deficit due to a 


trainees? 


best suited to trainees' needs 


lack of skills, knowledge, and 


- Do trainees know what goals 


and characteristics? 


attitudes? 


they have to achieve? 




- To what degree do 


- Is the training addressing the 




organizational factors 


trainees' problems and needs? 




influence the performance? 






- How difficult is it to perform 






the tasks? 






- How critical are eTors? 






- How frequent are errors made? 






- What are the learning 






characteristics of the trainees? 






- What is the level of skills, 






knowledge, and attitudes of 






trainees? 






- How much time can they 






spend on the intervention? 







Table 3: Evaluation questions at certain moments in the training cycle to gear training to the needs and 



characteristics of trainees 



2.2.3 Managers 1 and supervisors* involvement and commitment 

According to Gill (1989), managers often tolerate training because they view it either as a 
staff benefit or as a means to remedy a skill deficit for particular employees. Often they 
do not consider it as critical to the successful operation of the business. Therefore, 
managers often place less value on training and development programs than they do on 
other organizational functions. If managers do not show that they value training, trainees 
are less likely to apply the new skills and the training effects will be minimal. Therefore, 
the training professional should strive for involvement of managers and supervisors in 
the training process (Baldwin & Ford, 1988; Bowsher & Galbraith & Kaney, 1991; 
Caravan, 1991; Kaufman & Parry, 1991). 

It is the training professional's task to make managers and supervisors aware of 
what it means to change into new working habits and to develop their skills in leading 
their subordinates through change efforts. The training professional also has to undertake 
activities during the total training process to keep management and supervisors involved 
and he should ask their help in implementing a successful training program. 

Evaluation questions are lifted in table 4: 



Before 


During 


After 


- Who has indicated the need for 


- Is the supervisor showing 


- Is the supervisor supporting 


an intervention to reach the 


interest to the trainee in- 


the new behaviors that were 


organizational goals? 


between training sessions? 


learned in the training? 


- Do supervisors of the units 




- Are the employees given the 


concerned see the need of 




opportunity to apply new 


realizing the goals? 




skills, knowledge, and 


- What material and financial 




attitudes right away? 


support is already available 




- Are they given time to use 


that will help the trainee 




their new skills? 


apply the new skills, 




- How do trainees know that 


knowledge and attitudes? 




they are applying the skills, 


- Are supervisors preparing 




knowledge, and attitudes 


their employees to go to 




correctly? 


training? 







Table 4: Evaluation questions at certain moments in the training cycle for management i/ivolvement 
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3. Study of exemplary evaluation practices 

3.1 Questions 

Literature came up with a lot of suggestions for an effective use of evaluation. However, 
many of these suggestions were on the level of ideas and assumptions, and had not 
actually been validated by research or documented practice (the questions presented in 
chapter 2 were derived from the suggestions and are examples of what could be asked). 
Therefore, the main purpose of the second part of the study was to get practical and 
concrete examples of how evaluation for impact can be undertaken. 
The following questions were formulated: 

- What are examples of evaluation that had impact on a training or organizational level? 

- What evaluation activities have been carried out? 

- What was the purpose and focus of the evaluation? 

- What was the object of evaluation? 

- Who was involved? 

- When and where was evaluated? 

- What problems were encountered? 

- How were they solved? 

The study produced a wealth of examples of concrete successful procedures and 
instruments that training professionals have used in their training practice. Analysis of the 
data, using Hyperqual, resulted in general recommendations. These recommendations 
will be presented below. Because it would go beyond the scope of this paper to present 
all the practical examples for each of the recommendations, the recommendations will be 
illustrated by the information from one of the case studies. Therefore, first a short 
description of this case will be given. 

3.2 Example of a case context 

A case study was conducted in a fast growing company that sells medical products by 
telephone, as an intermediary between the manufacturer (the client) and the potential 
buyers in the medical field. If a client wants to bring his medicins under people's 
attention, a number of operators in the company start calling to different possible buyers, 
such as doctors, hospitals, psychologists, etc. Because the telephone is the only contact 
between the operator and a potential buyer, the operator should possess good 
communication skills that lead to a selling of the product. 



With the rnpid growth of the company, some time ago, the need was felt to get the 
work processes of the telephone operators more structured and controlled. Several 
operators were promoted to team leader and were supposed to monitor and coach their 
subordinate operators, and to assess their calls in order to give them suggestions for 
improvement, ultimately improving the company's services. Gradually, however, it 
became apparent that the team leaders were not assessing the phone calls in the same way 
and they were getting uneasy and not taking coaching seriously or not taking it the way 
that it was intended. 

To solve this problem, an external agency with close contacts to the company was 
hired to carry out a needs assessment. Based on problem areas that were detected in the 
needs assessment, it was decided that a training would be a good solution. In the 
training, team leaders learned what good selling skills are and how they could recognize 
them. They learned a standard approach for assessing an operator's call, indicating areas 
for improvement, and communicating this to the telephone operator in a coaching 
session. These coaching sessions fall apart in three parts: indicating what the good 
aspects of the call are, indicating areas of improvement, and giving strategies to improve. 

A lot of practice and evaluation has been built into this training. Not only training 
activities, but also the work of operators and team leaders are embedded in evaluation. 
Checklists are used to keep track of the quality of calls and quantitative measures are used 
to track progress and productivity. It is the company's experience that this evaluation is 
extremely helpful in planning and controlling the work process and quality. 

3.3 Findings 

3.3.1 Conduct a needs assessment 

The needs assessment is a critical part in the training design process. The needs 
assessment has two parts, the first aimed at organizational needs, the second at trainees' 
needs. In practice, these parts merge more or less and information may be gathered in 
conjunction, but a clear distinction helps the training professional conceptualize better. 

The goal of the first part is to identify what organizational effectiveness is and in 
what ways the current organization is falling short of being effective. Based on this first 
part of the needs assessment, the consultant should determine whether training is the best 
solution to solve (part of) the problem(s). 

If training seems to be the best solution, the consultant should gather more 
specific information regarding the training context, trainee characteristics, and aspects of 
trainees' jobs and tasks that need some adaptations. If information has been gathered well 
in the needs assessment, the consultant should be able to derive objectives for the training 
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easily. These objectives should, thus, reflect organizational goals and can be measured 
afterwards to see whether they have been reached. 



At a specific moment, the 'telephone company' wanted to determine the training budget for 
the next year and asked an external consultant, who was well-known in the company, to do 
a needs assessment to find out where they should spend money in order to get the biggest 
pay-back. In the $ 3000 needs assessment that followed, three kinds of activities were 
carried out: the external consultant had some interviews with the management team (4 
persons), had focus groups with all team leaders and a representative sample of telephone 
operators (15 in total), and assessed operator performance by observing calls and evaluating 
the coaching session between operator and his team leader. 

The main topics in the focus groups and interviews were: What is the strength of the 
company? Where on the road must the company be in a few years? What are the problem 
areas? What has to be improved? What are the priorities? What are the performance 
problems? Some follow-up questions were asked for each person specifically, depending on 
the findings of the focus groups and interviews. 

The problems and needs that came out of these activities were put on paper and 
presented to the management team. They were given the opportunity to prioritize what 
problems should be handled first, which was decided upon unanimously to be the team 
leaders' coaching. At this moment, the question became: Why hire a training consultant? Is 
training the solution? In a discussion between management team and consultant, it became 
clear thai the main problems lied in the performance of the team leaders, and that training 
would be a good solution. 

The content of the training was based on the problem areas that had been detected in the 
needs assessment and for each problem area, improvement rates were established, thus using 
the information as baseline data. 

When the training started, team leaders as well as the manager felt very committed, 
because the problems that were to be dealt with in the training, were those that were closely 
connected to their work and that they had indicated themselves . 

3.3.2 Design and deliver training iteratively 

An iterative approach to training design, as far as preconditions allow one to do that, 
turned out to be very effective in some cases. It allows the training professional to closely 
monitor whether the training is still geared to the needs of the organization and 
employees, and whether the training will meet or is meeting quality standards. Generally 
two kinds of iterative approaches were reported: the iterative design, and the iterative 
delivery. 

The first one means that, during the design process before the official training 
event, subsequent parts of the training design are evaluated. Examples are an evaluation 
of the training outline by trainers, or an expert appraisal on training content Another 
example is the pilot test. In the pilot test, a first or second version of the training is run on 
a small scale, with a representative sample of trainees, and information is gathered on 
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how it works, what the problems are, etc. This iterative approach assumes that experts, 
client, future trainees, and others involved, have time available at the moments the design 
needs to be evaluated. 

Second, when the training is ready to be delivered, the iterative approach is also 
very effective. Although the major outline of the training sessions should be clear when 
the training starts, and major problems should have been detected in earlier pilots, the 
delivery of the training program in small pieces with some time in-between sessions 
offers some latitude for the training professional. It allows the trainers to evaluate a 
session and use the information to better implement the next ser i,Ion. This holds as long 
as a next session will deal with the same topic or whether there is time available to deal 
with the same topic, should that be necessary. Also, it provides trainees time to practice 
on the job in-between sessions what they learned in the training, which enhances their 
skills. It also enables to come to the next session with specific questions and problems 
that can be dealt with. Another advantage of the delivery in small steps is that trainees do 
not need to be away from their jobs for a long time, assuming that the training takes place 
not far away from the job. 



The training, as given in the telephone company, had evolved from training for previous 
clients and had the standard outline ready. Before the training started, some content related 
examples were chosen from the team leaders' and operators' work, in close cooperation with 
the team leaders 1 manager, but that was the only adaptation that was made. Previous 
changes were aimed at increasing practicality, repetition and feedback. 

The training itself was delivered in 6 sessions, each 4 hours long, one week apart. As 
home-work, the team leaders applied the skills that they had learned in the training in the 
coaching sessions with their operators. Hereforc, they used a checklist that was handed out, 
discussed, and practiced in the training. After a few practice sessions and before the next 
training session started, the team leader had to show the manager real mastery of the skills, 
either by coaching the manager in a fake situation in which the manager played an operator, 
or by having the manager observe a real coaching session. In this way, it was made sure 
that a team leader was ready to attend the next session. 

Each training session that discussed important new skills, ended with a 5 till 15 
minutes evaluation. Trainees filled in an evaluation form and indicated (orally) the specific 
strengths of the session and identified areas for improvement (what aspects they needed to 
practice more) and how they would like to see that changed. These aspects were either dealt 
with by the manager in-between sessions, or were emphasized or practiced in the next 
session. Note that here also, the the evaluation starts from a positive point of view and 
asks for improvement strategies. 

After each session, the instructor and the manager discussed in a few minutes how the 
training was going, what aspects would have to be changed, added or deleted in the next 
session. They also discussed how the team leaders were doing and to whom and to what 
aspects the manager should particularly pay attention during the homework sessions. 
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3.3.3 Get supervisor and employee together before the training to discuss 
the training 

As had been read in literature, the commitment of supervisors is one of the most essential 
factors contributing to training impact. One of the ways in which management can show 
their commitment to their employees is to sit together before the training and make some 
appointments, regarding the application of learning and each party's responsibilities. In 
this way, the employee learns that he will be held accountable for the training and will 
have to apply newly learned skills, while the manager should guarantee that the time and 
materials needed will be available. Thus, these kind of sessions between supervisors and 
employees set standards for later evaluation of skill mastery. 



Usually, in the telephone company, team leaders and their manager meet once every week 
to check how everything is going. This did not change at the time of the training. In one of 
their weekly meetings, the manager explained each individual team leader why the 
management team had decided for a training. Because they had indicated problem areas in 
the needs assessment themselves, team leaders were aware of problems and the 'training 
news 1 did not come unexpected. All team leaders being informed, the goals for the training 
were outlined in a general meeting with all team leaders, manager, and training consultant. 
It was indicated what areas of coaching would be dealt with and what team leaders would be 
able L do after the training, under what conditions. 

Team leaders were told that they should pass an exam after the training. The passing of this 
exam would mean a prolonging of their job as team leader. If they failed, they would be 
given the possibility to try again after a while, but in the meantime, they would only work 
as operator. It was also enhanced that they would practice a lot in the training and in- 
between sessions, so that they would not take the exam unprepared. The checklists that 
were used in the training and coaching sessions would also be used in the exam. 



3,3.4 Follow-up on the training 

The goal of a follow-up is to reemphasize the content, the importance of skills 
application, and to hold the trainees and their supervisors accountable for demonstrating 
the skills. 

The moment at which a follow-up should be done must meet two conditions. 
First, it should be as soon after the training as possible to make sure that the employee 
will not go back to old behaviors. Second, the time frame after the training must be long 
enough to enable the employee to apply the new skills. 



In the telephone company, the follow-up is done in the weekly session between the team 
leader and the manager, in which the team leader has to show the manager mastery of the 
skills. Questions that are discussed in the session are: 



What were the most important aspects of the training? 

- What did the trainee learn in the training? 

- Can the trainee apply the skills on the job? 

How well was the employee able to apply the skills? 

- What factors hindered the employee to apply the skills on the job? 

Because these weekly sessions are part of the work routine and will remain when the 
training has stopped, the training will be enhanced for a long time. 



3.3.5 Store evaluation information in an (electronic) database 

A very useful approach for those organizations and companies who want to keep close 
control over the quality of the workprocesses or the training, is to build an (electronic) 
database in which evaluation can be stored regularly. The advantage of storing evaluation 
information of training and job aspects in the database is to enable the client or consultant 
to easily track whether things have changed over time in either positive or negative way. 
It is especially useful in those situations in which a slight underperformance would have 
major consequences. Because it is often hard to mere observe slight underperformance, 
the use of the computer is supposed to help detect this underperformance earlier on. What 
is gained (financially) by turning the problems down in an early stage, outweighs the 
costs that have to be invested in the installation of the system. In other situations, one 
should ask themselve whether the investments in the system will be returned positively. 

It might be worthwhile to input training evaluation information in a datafile, 
especially in those cases where training is part of a yearly offering. It will allow the 
professional and his client to easily draw conclusions regarding the quality and results of 
the training. This will increase the accountability of the training department in an 
organization. 



The telephone company has built an elaborate tracking system over years. 
Whenever the company gets an assignment from the client, a contract is made that specifies 
the minimum amount of products that will be sold within a specific period. For example, 
they may agree with the client to do 1000 calls and to end up with at least 200 orders 
within 3 months. These measures are translated into a number of complete calls per hour 
that each telephone operator has to reach, in order for the company to make profit. 

Each time a telephone operator is making a telephone call, he stores information about 
the call in the computer, for example who he has called, whether or not the product has 
been sold, what further appointments have been made, etc. 

One person in the company gathers the information from all operators into one file. He 
formalizes, formats, and prints the information. These prints - greensheets - that are pulled 
from the system each day before nine o'clock, give numeric information about what each 
individual operator did the day before. It provides information about the total hours the 
operator has been on the phone; the number of calls that have been made; the number of 
calls that were completed, which means that either the product is sold, or an appointment 
for a future call is made, or the telephone is out of order. The greensheets give also 
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impact 



information about the number of orders that were gotten, how accurate the call was, how 
many hours the operator has worked in total, and how many hours of over time were made. 

Based on this daily information, a team leader is able to see how well an operator is 
reaching the campaign's objectives. For instance, if someone has completed three calls per 
hour instead of five, as written in the campaign contract, this operator performed under 
average that day. Using the sheets from some successive days, the team leader is able to 
identify whether or not this performance is structural and whether something should be 
done about it. The quantitative information from the greensheets is used in conjunction 
with the qualitative coaching and monitoring skills that the teamleader learned in the 
training: when the greensheets show that an operator is performing under level, tl £ team 
leader discusses this with the operator and tries -by observing one of his calls- to d- -ect the 
problem areas and to give strategies for improvement. In this way, a team leader is not 
only able to detect that an operator has some problems with calling, but also to indicate 
what the areas of weakness are and to give directions for improvement. 

The information from this system is used every day, and adjustments to the operator's 
behavior can therefore be given in an early stage, increasing the chance of attaining the 
campaign's objectives. 

On a weekly level, the spreadsheets give information about the progress of a campaign. It 
enables the campaign manager to check the iikeliness of reaching the campaign goals. If 
necessary, adjustments can be made in an early stage. The use of these spreadsheets also 
allows the company to improve from campaign to campaign. For example, if the average 
sale score on one campaign was 30%, although the goal was 50%, probably the goals were 
set wrong. They can bear this in mind in the next campaign. The spreadsheets also enable 
the company to get an idea of the 'best sellers'. In one of the meetings I attended, an extra 
telephone operator had lo be chosen for a new campaign. The group used the information 
from the greensheets to select a person. 

4. Discussion 

In the study on exemplary evaluation practices, concrete examples were found of 
evaluation approaches that were experienced to help get training impact by training 
professionals and other persons concerned. By looking in 'the kitchen' of evaluation, 
several examples of useful procedures and instruments were found. In this paper, we 
have tried to sketch the procedures and instruments of one exemplary evaluation practice 
as an illustration of how evaluation can reach impact. In the telephone company, 
evaluation proved to be a powerful tool in getting the team leaders on the right level of 
coaching, and it also served as a means to keep close control over work processes of 
individual telephone operators. 

The American findings that were translated into recommendations were given to a 
few Dutch experts for their comments and reactions. A lot of the approaches and 
instruments that were described were judged to be useful in the Dutch situation as well. 



However, the Dutch interviewees had two general reservations that went beyond the level 
of specific instruments. 

First, it was the general opinion of the Dutch interviewees that the training 
consultant can only be held responsible for the quality of the training materials and 
training event itself, not for more, not for less. The training should be of outstanding 
quality and should enable trainees to learn the skills they need to learn. Although they 
valued supervisor involvement as important and the creation of an environment in which 
new learned skills can be applied very advisable, they considered it the responsibility of 
the supervisor. The importance of supervisor involvement can be communicated to the 
supervisors, but the degree of their involvement depends on themselves. Although the 
American interviewees also agree that skill application is the supervisors' responsibility, 
they claim that it is the training consultant's responsibility to try very hard to get 
supervisors involved, Examples of practices in which the training consultant's effort 
really got supervisors involved which led to positive results, makes this claim even 
stronger. 

A second difference was found in the ways in which evaluation is and can be 
used. Whereas, in the exemplary practices, evaluation seems to be fairly integrated in the 
work and training processes, Dutch interviewees argued that it is almost impossible to 
realize this amount of evaluation in Dutch organizations. Evaluation is often seen as a 
threatening activity and, therefore, the training consultant should use it carefully and 
sparingly. An evaluation that goes beyond measuring direct training effects is difficult for 
the training consultant to realize. For example, a document analysis at the start of the 
design process to detect problem areas and gather base line data for the training was used 
and experienced in a few of the American projects as an efficient method, because the 
people involved did not have to invest large amounts of time in interviews. The Dutch 
interviewees argued that, generally, document analysis is beyond the scope of the training 
consultant as it may reveal confidential information. Also, evaluation situations other than 
manager-employee, such as peer evaluation, are difficult to realize. 

Several reasons can be found for these differences between the American findings 
and the Dutch remarks, first of all, one should remember that the study in the States was 
aimed at finding examples from successful evaluation practices, as a realization of the 
suggestions in literature. This means that situations were found in which the project 
variables were such, that specific evaluation procedures and instruments were not only 
possible, but also considered useful. This does not mean that they would be possible and 
successful in every single situation. Although the recommendations themselves aimed at a 
certain level of generalization, they were illustrated with these specific one-case 
successful examples. On the other hand, the comments from the Dutch interviewees were 
more from a general point of view, rather than specific: to what respect would a certain 
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procedure or instrument be useful in a typical Dutch situation. Because of this broader 
view, it seems logical that they showed some reservations. 

A second reason for the differences that were found in the US and Dutch part, can 
be due to a difference in evaluation tradition. In the Netherlands in general, it seems that 
evaluation is still seen as an activity that primarily aims at detecting failures and errors. As 
such, it is an activity that people are kind of afraid of. The example from the telephone 
company, however, shows that evaluation can have a positive nature as well. Here, the 
purpose of the evaluation was not to detect only negative aspects: each evaluation session 
started from a positive point of view: what were the good aspects of an operator's call, 
what were the good aspects of team leader's coaching, what were the good aspects of a 
training session. After determining the good aspects, the 'evaluator' indicated areas of 
improvement and, especially, strategies for how these improvements could be made. 
Used in this way, evaluation lays a constructive basis for change and improvement and is 
probably one of the reasons that this project was judged as a successful one: not the mere 
use of evaluation, but rather the way in which it has been used caused the success. 

A third related reason for the differences seems to be grounded in what can be 
called an organizational culture difference. It seems that, in the Dutch organizations, there 
are fixed role patterns, that are hard to be broken: the role of the training consultant is to 
train, the role of the supervisor is to supervise, entering each other's territory is not 
allowed. It is not only that supervisors do not want training professionals to cross their 
paths, it even seems that training consultants themselves do not feel the need to change in 
a more collaborate direction. It seems to relate with one's view of what training is and 
should do: as long as training is regarded as a goal in itself, this clear role distinction can 
be held alive. The current trends in the organizational field, however, -that are also 
present in Dutch organizations- force us to regard training as a means to make the 
organization more effective, rather than a goal in itself. The examples from the US 
showed that training consultants need to become more flexible. That evaluation can add 
value in this process gets clear from the example of the telephone company. 
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